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INTRODUCTION 


Birenbaum  ^  Tatsuoka  (1980)  and  Tatsuoka,  et.  al.  (1980)  found 
quite  a  number  of  "bugs",  57,  In  addition  and  subtraction  problems  of 
signed  numbers.  These  bugs  are  driven  by  misconceptions  or  Incomplete 
knowledge  at  various  components  of  procedural  tasks  and  resulted  in 
erroneous  rules  of  operation  for  carrying  out  computational  problems. 

These  rules.  Including  the  right  one,  are  represented  by  the 
responses  to  a  particular  set  of  Items  because  each  rule  generates  a 
unique  set  of  responses  to  the  Items.  For  example,  the  rule  that  adds 
the  absolute  values  of  two  numbers  and  takes  the  sign  of  the  first 
number  in  the  answer  yields  responses  +15,  -15,  and  -19  to  the 
problems,  12  +  -3,  -3  +  12  and  -14  +  -5,  respectively.  But  the  right 
rule  produces  responses  +9,  +9  and  -19  to  the  same  problems.  By 
selecting  an  appropriate  set  of  Items,  the  two  rules  can  be  identified 
by  examining  the  response  vectors  (+15,  -15,  -19)  and  (+9,  +9,  -19) 
respectively.  Indeed,  some  error  diagnoses  are  derived  by  human 
Intuition  from  the  examination  of  responses. 

Bro\fli  &  Burton  (1978)  have  developed  computerized  diagnostic  models 
of  procedural  skills  In  arithmetic  computations.  Their  models  provide 
an  identification  of  mistakes  In  the  procedural  network.  Tatsuoka,  et 
al.  (1980)  introduced  an  "error  vector  system"  that  has  a  power 
equivalent  to  the  procedural-network  approach.  The  error  vector  sysfem 
begins  with  the  decomposition  of  a  computational  task  of  two  signed 
numbers  Into  component  procedures,  obtaining  the  absolute  values  and 
obtaining  signs  for  answers.  Thus,  the  two  subprocesses  are  represented 
by  two  error  vectors,  absolute  value  and  sign  error  vectors,  whose 
components  correspond  to  all  possible  alternative  operations. 

The  mechanism  of  diagnosing  the  misconceptions  possessed  by  a 
student  starts  first  by  generating  a  binary  vector  of  erroneous  and 
correct  actions  taken  by  the  student.  Thus,  several  different  binary 
vectors  (according  to  the  number  of  component  procedures  necessary  for 
carrying  out  the  task)  will  be  generated  from  the  student's  responses  to 
the  Items.  In  the  case  of  signed-number  arithmetic,  the  two  binary 
vectors  that  represent  "getting  the  absolute  values"  and  "choosing  the 
signs  In  the  answers"  will  be  generated.  The  matrix  product  of  the  two 
binary  vectors  will  determine  a  consistent,  complete  error  committed  by 
the  student.  If  any. 

Glaser  (1981,  p.  10)  states  that  "An  Important  skill  of  teaching 
is  to  Identify  the  nature  of  the  concept  or  rule  that  the  student  Is 
employing  that  governs  his  performance  In  some  systematic  way,  (In  most 
cases,  the  student's  behavior  Is  not  random  or  careless,  but  Is  driven 
by  some  underlying  misconceptions  or  by  incomplete  knowledge)."  The  same 
results  were  observed  by  a  series  of  experimental  data  conducted  by  the 
author  (1981). 

When  the  student  applies  his/her  rule  consistently  for  obtaining 
absolute  values  but  Inconsistently  for  taking  In  the  signs  of  the 


answers,  Chen  incomplece  rules  will  be  the  outcome  of  error  analysis. 
Birenbaum  (1981)  lists  all  the  erroneous  rules  she  found  by  examining 
students'  response  sequences.  Chaiklin  &  Greeno  (1981)  describe  cognitive 
backgrounds  for  these  rules.  Neither  the  computerized  approaches  of  Brown 
Burton  (1978)  nor  that  of  TaCsuoka,  et  al.  (1980)  can  diagnose  such 
incomplete  rules;  in  other  words,  Inconsistent  responses  in  at  least 
one  of  the  procedural  components  makes  their  task  difficult. 

Some  rules  may  be  incomplete,  but,  as  long  as  they  are  consistent, 
their  nature  of  error  can  be  characterized  by  assessing  the 
"seriousness".  Some  errors  are  due  to  misconceptions  originating  at  the 
basic  understanding  of  negative  numbers,  while  other  errors  are  caused 
by  minor  mistakes.  The  former  type  of  error  may  be  harder  to  correct, 
while  the  latter  case  will  be  easy  to  correct.  It  can  be  said  that  the 
serious  errors  are  caused  by  misconceptions  originating  at  the  upper- 
level  nodes  in  the  procedural  network,  if  procedural  components 
are  hierarchically  related. 

This  study  describes  an  attempt  to  quantify  the  degree  of 
"seriousness"  of  various  error  types  (or  erroneous  rules  of  operations) 
and  investigates  their  relationships  to  the  future  performance  of  the 
student.  The  development  of  a  methodological  tool  will  be  discussed  in 
conjunction  with  the  error  analysis  performed  on  signed-number  addition 
problems.  The  method  will  be  applicable  to  other  arithmetic, 
mathematics  and  science  problems. 


CLASSIFICATION  OF  RULES 


Procedural  representation  of  solving  a  given  problem,  such  as 
subtraction  problems  In  whole  numbers  (Brown  &  Burton,  1978),  addition 
of  two  fractions  (Greeno,  1976)  and  signed-number  arithmetic  (Birenbaum 
&  Tatsuoka,  1980),  has  become  a  popular  way  to  express  the  complex 
relationships  among  various  procedural  components.  In  order  to  achieve 
the  goal,  one  oiust  perform  each  subprocedure  correctly  and  go  to  the 
next  component.  All  necessary  procedural  steps  must  be  completed 
successfully.  If  a  student  falls  to  follow  a  right  procedural  step,  and 
does  not  perform  all  subprocedures  correctly,  then  the  outcome  is  likely 
an  erroneous  rule  ot  operation  or  some  mixture  of  erroneous  rules. 

Tatsuoka  et  al.  (1980)  developed  a  rule-diagnostic  model,  termed 
the  "error  vector  system”  for  signed-number  addition  and  subtraction 
problems.  They  decomposed  an  operation  of  computing  these  problems  into 
two  component  procedures  and  represented  each  component  procedure  by  an 
error  vector:  one  for  absolute  value  and  the  other  for  sign.  The 
elements  of  the  two  vectors  are  prospective  erroneous  rules  of  operation 
that  may  be  obtained  by  a  close  examination  of  component  procedures. 
Descriptions  of  the  rules  are  given  in  Table  1. 
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Insert  Table  1  about  here 

As  can  be  seen  in  Table  1  ,  the  absolute  value  vector  is  composed  of 
seven  elements,  and  the  sign  vector  of  nine. 

For  example,  suppose  a  student  subtracted  the  two  numbers  in  the  task, 
-5  +  -3,  and  took  the  sign  of  the  larger  number  in  the  answer.  Then 
this  erroneous  rule  of  operation  is  expressed  by  (13,21).  The  right 
rule  of  operation  is  designated  by  (11,21)  for  addition  problems. 

The  matrix  product  of  the  two  error  vectors  will  have  63  events 
that  correspond  to  different  rules  of  operation.  In  general,  there  are 
several  error  vectors,  each  of  which  represents  a  component 
procedure.  A  cartesian  product  of  these  error  vectors  will 
determine  various  erroneous  rules  of  operation. 

A  rule  is  said  to  be  complete  if  all  component  procedures  are 
consistently  performed  by  some  systematic  behavior.  In  other  words,  the 
responses  to  the  items  generated  by  a  complete  rule  will  be  uniquely 
determined  in  the  cartesian  space  of  the  error  vectors.  If  at  least  one 
of  the  component  procedures  is  not  consistent  then  a  rule  is  said  to  be 
incomplete .  An  incomplete  rule  cannot  produce  a  unique  set  of  responses 
to  the  items.  One  of  the  popular  errors  in  signed-number  arithmetic  is 
"taking  the  right  signs  in  the  answers  but  absolute  values  are  taken 
inconsistently."  With  such  a  partially  right  rule,  the  diagnostic 
models  cannot  determine  and  state  the  rule  completely.  Table  1  provides 
the  error  code  (21)  with  this  incomplete  rule.  Similarly  other 
Incomplete  rules  will  be  designated  by  two-digit  numbers  between  11  and 
19  or  21  and  29  according  to  which  component  procedure  is  consistent. 

Table  2  summarizes  the  number  of  complete  and  incomplete  rules 
recovered  by  the  error-vector  system  for  a  20-item  computation  test. 
Although  Table  2  shows  the  erroneous  rules  for  only  addition  problems, 
those  of  subtraction  problems  have  similar  results  (K.  Tatsuoka,  1981). 

Insert  Table  2  about  here 

The  test  was  administered  to  the  seventh  graders  at  a  local  junior  high. 
About  70%  of  290  sets  of  responses  to  five  addition  problems  are 
successfully  diagnosed  by  the  error-vector  system.  The  numbers  of 
complete  and  Incomplete  rules  are  about  the  same,  but  the  distrlbutioii 
of  frequencies  over  the  errors  is  far  from  uniform.  Fifty-five  sets  of 
responses  to  five  addition  items  demonstrated  the  right  rule  (11,21),  20 
and  14  response  sets  are  yielded  by  the  Incomplete  right  rules  (11)  and 
(21),  respectively.  The  incomplete  rules  (13)  and  (24)  have  frequencies 
of  24  and  23,  which  are  about  8%  of  the  total  290  response  sets.  Other 
erroneous  rules  such  as  (12)  and  (22)  are  rare. 

Investigation  on  the  differences  in  the  frequencies  of  error  types 
may  be  very  important  in  understanding  students'  states  of  learning  and 
improving  the  quality  of  teaching. 


4 


Table  1 

Decoding  of  the  Basic  "Error-Vector  System"  of  Slgned- 
Nunber  Operations  for  Two  Numbers 

addition  problems  such  as 

-L+S  (*assume  j  L|  >  |  S | 


Error  Type 

Description  of  the 

operation 

11 

If  the  signs  of  two  numbers  are  same, 

then 

|L(+|S( 

If  the  signs  of  two  numbers  are  different, 

then  |l| 

Absolute  Value 

12 

Always  |l|+|s|,  adding  the  absolute  values 

of  L,  S. 

operations 

13 

Always  1l|-|s1 

14 

Opposite  of  the  right  operation 

15 

If  the  sign  of  the  first  number  is  +, 

then 

iLl+ISl 

If  the  sign  of  the  first  number  is  -, 

then 

iLl+lsl 

16 

If  the  sign  of  the  second  number  is  + 

,  then  |l|+1S| 

If  the  sign  of  the  second  number  is  - 

,  then  !l1-|S| 

17 

If  the  sign  of  the  first  number  is  + 

then 

iLl-lSl 

Otherwise  opposite 

21 

Sign  of  the  larger  number 

22 

Sign  of  the  smaller  number 

23 

Always  + 

Taking  the 

24 

Always  - 

sign  of 

25 

Sign  of  the  first  number 

answers 

26 

Sign  of  the  second  number 

27 

Sign  of  the  product  of  the  two  numbers 

28 

Change  the  sign  of  the  second  number,  then  apply  21 

29 

Change  the  sign  of  the  first  number,  then  apply  21 

5 


Table  2 

Frequencies  and  Error  Types  Recovered  by  the  Error-Vector 
System  in  Testl,  Addition  Problems  of  Signed  Numbers 


Incomplete  Rule 

Frequency 

Complete  Rule 

Frequency 

11 

20 

11,21 

55 

12 

2 

12,23 

3 

13 

24 

12,24 

5 

21 

14 

12,25 

3 

22 

3 

13,21 

17 

23 

15 

13,22 

1 

24 

23 

13,23 

5 

25 

6 

13,24 

10 

26 

5 

13,25 

3 

16 

1 

13,25 

3 

Total* 

113 

Total 

_ ins _ 

*72  Response  Patterns  are  either  inconsistent  or  generated  by 
rules  besides  those  in  Table  1 
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ASSESSING  THE  SERIOUSNESS  OF  ERRORS 
Are  Two  Rules  Independent? 

Two  different  rules  may  represent  different  dimensions  and  hence 
may  be  statistically  independent.  But  others  might  not  be  independent 
from  one  another.  Of  course,  errors  are  outcomes  of  misconceptions  or 
incomplete  knowledge  somewhere  in  a  series  of  complex  procedures.  If  a 
student  does  not  understand  the  difference  between  operation  signs  and 
the  signs  of  numbers,  then  it  is  very  unlikely  that  his/her  errors  are 
derived  by  certain  systematic  mistakes  such  as  "taking  the  sign  of  the 
first  number  in  his/her  answer."  His/her  erroneous  rule  will  more 
likely  by  derived  from  an  ignorance  of  the  concept  new  to  signed-number 
arithmetic,  so  it  may  be  like  "the  sign  of  the  answer  is  always  minus 
for  subtraction  and  plus  for  addition  problems."  These  two  error  types 
will  probably  not  appear  together  in  the  responses  by  a  single  student 
to  the  test  items. 

Table  3  shows  the  list  of  rules  committed  by  the  students  which  are 
obtained  from  the  error  analysis  on  the  same  test  described  earlier. 


Insert  Table  3  about  here 


The  52-ltem  test  was  designed  to  examine  stability  of  erroneous  rules  of 
operation  so  that  four  13-ltem,  parallel  subtests  are  repeatedly  given 
in  the  test.  The  rule  in  the  first  pair  of  parentheses  stands  for  the 
first  subtest,  and  the  fourth  for  the  fourth  subtest  in  the  52-item 
test.  Error  patterns  are  examined  separately  for  addition  problems  and 
subtraction  problems  in  each  subtest.  Table  3  provides  information 
obtained  from  analysis  of  addition  probems.  The  Greek  letter  9  in  the 
last  column  of  the  table  denotes  a  student  whose  verbal  ability  is  high, 
within  the  top  16%  of  the  students. 


Insert  Table  4  about  here 


Since  each  student  took  the  four  parallel  subtests,  the  frequency 
Cables  of  rules  for  the  first  and  second  subtests  are  obtained 
separately  and  -  tests  for  Independence  of  rules  are  carried  out. 

The  first  test  has  ~  value  of  22.57  with  p  =  0.0002  and  the  second 
test  has  X*  ”  18.62  with  p  =*  0.0009  .  The  results  suggest  that  the  rules 
are  not  statistically  Independent.  Thus,  the  students  who  have  mastered 
the  right  operation  in  terras  of  the  absolute-value  operation  tend  to 
achieve  the  right  operation  for  taking  the  signs  in  their  answers. 

Those  who  answered  inconsistently  with  regard  to  taking  the  signs  in  the 
answers  also  have  a  tendency  toward  having  inconsistent  absolute  value 
operations. 

A  close  investigation  of  changes  in  error  types  over  the  four  subtests 
reveals  that  there  are  apparent  systematic  relationships  in  shifting  to  one 
rule  from  another,  although  quite  a  number  of  students  repeatedly  used 


List  of  Krror  Types  in  the  First  Test  of  the  lanuary  Experiment 
for  those  Students  UHiose  Responses  are  Highly  Consistent 
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+Inconsistent  responses 


Addition  Problems 


22.24  p  =  0.0002  X  =  26.69  p<  0.00005 
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their  own  rule  consistently  In  order  to  respond  to  the  Items  across  the 
four  subsets  In  the  test.  Incomplete  and  complete  rules  often  appear 
together  when  the  former  l^  .1  partial  rule  of  the  latter.  For  example, 
the  pattern  of  student  1  ,  0  ( 1 3 ,24) ( 1 3, 24) (24) ,  Is  a  typical  example  of  the 
situation.  The  second  and  third  are  complete  rules,  while  the  fourth  is 
an  Incomplete  rule.  This  student's  state  of  learning  is  obviously 
around  the  wrong  rule  (13,241.  His  rule  Is  always  to  take  the  difference 
of  absolute  values  of  the  two  numbers,  the  smaller  one  from  the  larger 
one,  and  using  minus  signs  in  the  answers.  He  did  not  apply  an  appropriate 
rule  of  operation  for  different  types  of  problems  and  also  failed  to 
discriminate  the  operation  sign  of  the  problems  from  the  signs  of  the 
numbers.  It  is  an  understandable  mistake  because  he  has  not  yet  studied 
subtraction  skills  when  he  took  the  test. 

The  second  kind  of  pattern  in  Table  3  is  that  of  Incomplete  and 
complete  rules  appearing  together  but  the  Incomplete  one  is  not  a 
partial  rule  of  the  complete  rule.  For  example,  student  54  has  the 
pattern,  (13,24)  0  (21)(25).  The  rule  related  to  the  sign  operation 
changes  three  times:  24,  21  and  25,  that  is,  "always  taking  a  minus 
sign",  "taking  the  sign  of  the  larger  number"  and  "taking  the  sign  of 
the  first  number".  It  can  be  said  that  this  student  is  not  confident 
with  her  rule  of  taking  the  signs  in  answers,  so  she  chooses  the  signs 
arbitrarily  by  trial  and  error.  Her  stability  over  the  four  subtests  is 
low.  Thus  It  Is  difficult  to  locate  her  rule  from  the  test,  at  least 
from  the  present  information  about  her  performance  on  the  test.  It 
might  be  necessary  to  give  her  more  items  so  as  to  determine  her  rule. 

The  third  kind  of  pattern  is  expressed  by  the  following  example: 
students  41,  42  and  58  always  subtracted  two  absolute  values  and  took 

the  sign  of  the  larger  number  in  the  answers.  Their  pattern  is 

(13,21)(ll,2l)(l  1,21)(11,21).  Their  rules  reached  the  right  rule  of 
operation  at  the  second  try.  The  change  of  rules  involves  two  complete 
rules,  (13,21)  and  (11,21). 

The  fourth  kind  includes  0  ,  which  means  the  error  vector  system 

could  not  Identify  the  rules  used  by  students.  So  we  characterize  the 

rules  as  "inconsistent  rules"  as  far  as  they  can  not  be  determined  by 

the  present  error-vector  system. 

The  errors  resulting  from  misunderstanding  one  or  two  basic 
concepts  can  be  said  to  be  serious  because  it  is  not  easy  to  remove 
these  errors.  The  students  have  to  study  the  basic  concepts  which  are 
usually  presented  at  an  earlier  part  of  instruction.  The  third  kind  of 
error  type,  switching  from  (13,21)  to  (ll,2l)  might  be  easy  to  correct 
because  (13,21)  came  from  ignoring  the  difference  between  skills  of  the 
-L  +  -S  and  -L,  +  S  'ypes.  If  the  two  signs  of  the  numbers  are  the  same, 
the  absolute  value  for  answers  must  be  obtained  by  adding  two  numbers 
Instead  of  subtracting  them.  The  student  knows  the  difference  between 
operation  signs  and  the  sign  of  a  number.  It  can  be  said  that  he/she 
just  failed  to  distinguish  a  -h  +  -5  type  skill  from  a  -L  +  S  type 
skill.  Indeed,  (13,21)  and  (11,21)  appeared  together  in  one  response 


pattern  quite  often  (11  times).  In  this  sense  (13,21)  is  not  serious 
and  is  easy  to  correct. 


QUANTITATIVELY  ASSESSING  THE  SERIOUSNESS  OF  ERRORS 
Definition  of  Seriousness 

One  approach  to  quantifying  the  seriousness  of  error  types  is  to 
assess  the  extent  to  which  a  rule  vitiates  the  steps  necessary  to  achieve 
a  given  task  successfully.  In  other  words,  each  error  must  have  reasons 
as  to  why  and  where  it  occurred.  Some  errors  are  due  to  students' 
careless  mistakes,  but  others  —  quite  a  number  of  them  —  are  due  to 
the  lack  of  understanding  at  some  level  of  the  procedural  steps.  If  the 
errors  were  caused  by  misunderstanding  at  some  deeper  level  of  the 
procedure,  then  it  will  not  be  easy  to  remove  them.  If  the  errors  were 
caused  for  two  or  more  compounded  reasons  such  as  missing  several 
procedural  steps  to  arrive  at  the  right  answer,  then  this  is  more 
serious  than  an  error  caused  by  missing  just  one  step.  If  the  error 
occurred  at  steps  close  to  the  end  of  the  procedural  network,  then  this 
error  might  be  easy  to  correct;  thus,  it  is  less  serious.  In  order  to 
assess  the  seriousness  of  errors  in  terms  of  the  procedural  network,  we 
must  know  the  network  first.  Regarding  the  networks,  since  there  is 
much  good  literature  written  by  cognitive  psychologists  (e.g.,  Gregg, 
1976;  Resnick,  1976;  Greeno,  1976),  the  author  assumes  that  a  suitable 
task  analysis  has  already  been  achieved  and  hence  we  know  our  network. 

Suppose  everybody  knows  what  negative  numbers  are  and  can 
discriminate  them  from  positive  numbers.  Thus,  knowing  the  negative  and 
positive  numbers  is  required  as  a  prerequisite  of  the  tasks  of  carrying 
out  the  operation  of  addition  problems.  Then  the  next  step  is  that 
students  must  discriminate  operation  signs  from  the  signs  of  numbers. 

The  third  step  is  the  identification  of  the  type  of  addition  task,  +[  ] 

+  ]»  ~[  ]  f  ]  or  [  ]  +  [  ],  ~[  ]  +  “[  ]•  For  tasks  in  which  the 

two  numbers  have  different  signs,  the  right  absolute  values  of  answers 
are  gotten  by  subtracting  the  absolute  value  of  the  smaller  number  from 
the  larger  number,  while  the  other  tasks  require  adding  the  absolute 
values  of  the  two  numbers.  After  taking  the  right  operation  of  absolute 
values,  the  students  have  to  choose  the  right  sign  for  their  answers, 
which  Involves  taking  the  sign  of  the  larger  number  in  both  cases. 

By  representing  all  component  procedures  for  carrying  out  the 
addition  problems  properly.  Figure  1  gives  a  graphical  representation  of 
the  procedural  network.  Note  that  the  number  at  the  upper  left  corner 
of  the  rectangle  for  each  node  represents  the  level  of  that  task 
in  the  hierarchy. 


Procedural  Steps  Conformity  Index  (PSCI) 


TaCsuoka  &  Tatsuoka,  (1980)  Introduced  a  new  index,  the  Norm 
Conformity  Index,  which  measures  the  degree  of  conformity  of  a  response 
pattern  to  a  criterion  order  for  each  examinee.  The  criterion  order  is 
usually  determined  by  some  external  consideration  such  as  adopting  the 
order  of  item  difficulty  in  a  test  administered  to  a  certain  group,  or 
that  of  a  national  norm  group.  We  now  conlder  possible  orderings  of 
component  procedures  Instead  of  items.  Since  the  base  order  is  again 
arbitrary,  the  order  determined  by  the  component  hierarchy  is  one  of 
natural  choice.  The  sequence  from  1  through  5  in  Figure  1  will  be  taken 
as  the  base  order.  These  procedures  thus  ordered  will  be  referred  to  as 
procedural  levels  hereafter. 

A  rule  can  be  checked  with  each  procedural  level  to  see  whether  or 
not  it  is  based  on  a  misconception  or  incomplete  knowledge  occurring  at 
that  level.  By  assigning  1  or  0  to  all  the  procedural  levels,  the  rule 
will  be  represented  by  a  binary  vector.  The  vector  will  be  referred  to 
as  a  process  pattern  hereafter. 

For  example,  if  the  item  -14  +  -5  has  been  responded  to  by  the 

rule  (13,21),  the  process  pattern  will  be  (11001),  by  the  following 
reasoning.  The  first  element  is  assumed  to  be  1  for  the  relevant  grade. 
Since  Incomplete  rule  21  implies  that  a  student  knows  the  difference 
between  the  sign  of  the  numbers  and  the  operation  sign,  the  second 
element  of  the  process  pattern  will  be  1.  Using  partial  rule  13  (always 
doing  |L  I  -  I  S|)  implies  inability  to  Identify  the  task  type;  hence  the 
third  and  fourth  elements  of  the  process  pattern  are  0.  The  fifth 
element  is  obviously  1. 

The  test  administered  to  the  seventh  graders  contains  five  addition 
tasks:  (1)  L  +  -S,  (2)  -S  +  L,  (3)  -L  +  -S,  (4)  S  +  -L  and  (5)  -L  +  S. 
For  each  of  these  five  tasks  a  process  pattern  is  defined  by  any 
given  rule  of  operation.  Those  corresponding  fifteen  of  the  erroneous 
rules  obtainable  from  Table  1  are  summarized  in  Table  6. 


Insert  Table  6  about  here 


Procedural  Steps  Conformity  Index  (PSCI)  is  defined  as  a  linear 
transform  of  the  following  ratio:  the  number  of  transpositions  needed  to 
transform  a  process  pattern  vector  to  its  reverse  Guttman  scale,  divided  by  the 
total  number  of  possible  transpositions.  Let  us  denote  the  former  by  U 
and  the  latter  by  U.  Then: 

PSCI  -  1  -  2Ua/U 

The  total  number  of  I's  in  a  process  pattern  shows  the  number  of 
procedural  levels  a  student  followed  correctly.  The  average  of  the 
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Table  6 

Process  Pattern  Vectors  Assiciated  with  Task  Components  for 
Five  Item  Types,  1.  L+-S,  2.  -S+L,  3.  -L-t— S, 

4.  S-t — L  and  5.  -L+S. 
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total  number  of  I's  across  the  five  tasks  will  be  called  the  mean 
procedural  score  (MPS)  associated  with  a  rule.  The  MPS  and  the  average 
of  the  five  PSCI  values  for  each  rule  are  considered  as  Indicators  of 
the  seriousness  of  the  mistake. 

Table  7  summarizes  the  values  of  these  Indices  for  the  same 
examples  as  given  In  Table  6.  Figure  2  Is  a  plot  of  the  erroneous  rules 
of  operation  listed  In  Table  6. 


Insert  Table  7  about  here 


Insert  Figure  2  about  here 

From  Figure  2,  It  Is  obvious  that  the  rules  are  classified  Into  two 
groups,  A  and  B.  The  members  In  Croup  A  cluster  at  the  upper  right 
corner  of  the  plane  while  those  In  Group  B  fall  In  the  extreme  left  side 
of  the  plane.  The  right  rule  (11,21)  Is  located  at  the  top-right  corner 
and  It  belongs  to  Group  A.  Therefore  the  rules  In  Group  A  have  much 
higher  proximity  to  the  right  rule  than  those  In  Group  B.  Some  of  the 
notable  characteristics  of  the  rules  In  Croup  A  are  that  (1)  they  follow 
more  procedural  levels  correctly  than  the  others,  and  (2)  they  tend  to 
have  higher  values  of  the  Procedural  Steps  Conformity  Index.  Further, 
Table  6  Indicates  the  rules  In  Group  B  missed  both  of  the  Important 
concepts  at  levels  2  and  3  but  the  rules  In  Group  A  followed  them 
correctly  and  missed  both  or  either  of  levels  4  and  5. 


Changes  In  Errors  as  Learning  Stages  Advance 

The  bugs  classified  In  Croup  B  originate  In  the  misunderstanding  or 
Ignorance  of  one  or  more  basic  levels  of  the  procedural  task.  A  question 
arises  whether  or  not  these  error  types  have  certain  relationships  with 
verbal  ability  because  the  Instruction  given  before  the  test  used  verbal 
stories,  the  Postman  Stories  (Oavis,  1963),  to  teach  signed-number 
computations.  A  second  question  Is  whether  or  not  frequencies  of  these 
erroneous  rules  depend  on  the  learning  stages  o  the  students.  Which 
rules  will  be  observed  more  often  In  data  as  the  students  approach  their 
mastery  level?  The  answers  to  these  questions  can  be  found  only  by 
empirical  investigation. 


Insert  Table  R  about  here 


The  Stanford  Verbal  Test  was  administered  to  the  same  subjects  as 
those  In  Table  7  and  Figure  1.  Their  sblllty  measures  were  obtained  by 
applying  Item  Response  Theory  (Lord,  1980;  Lord  &  Novlck,  1968).  In 
Table  2,  the  students  whose  estimated  ability  level  for  the  test  were  In 
the  top  I6t  (which  Is  9  ^  1)  were  marked  by  6.  Table  8  Indicates  that 
64X  of  the  rules  used  by  O-students  belong  to  Croup  A,  and  only  16X 
belong  to  Group  B.  The  trend  of  percentages  Is  reversed,  however,  for 


Table  7 

The  Summary  of  the  Means  of  the  Number  pf  Steps  Followed  by  each 
Error  Type  In  the  Test  Tree  and  of  the  Values  of 
Process  Tree  Conformity  Indices 


Error  Types 

Mean  of  Steps 

Mean  of  PTCI 

(11) 

4.0 

1.00 

(12) 

1.2 

.87 

(13) 

1.8 

.47 

(21) 

4.0 

.50 

(22) 

3.2 

.90 

(23) 

1.0 

1.00 

(24) 

1.2 

.80 

(25) 

1.6 

.40 

(26) 

1.6 

.40 

(11,21) 

5.0 

1.00 

(11.22) 

4.2 

1.00 

(11,24) 

3.4 

1.00 

(11,25) 

4.6 

1.00 

(11.26) 

4.6 

1.00 

(12,21) 

4.2 

0.60 

(12.22) 

3.4 

1.00 

(12.23) 

1.2 

0.87 

(12,24) 

1.4 

0.73 

(12,25) 

1.8 

0.33 

(12,26) 

1.6 

0.40 

(13.21) 

4.6 

.8  7 

(13,22) 

4.0 

0.70 

(13.23) 

1.8 

0.47 

(13,24) 

2.0 

.26 

(13,25) 

2.4 

0.00 

(13,26) 

2.4 

0.00 

Table  8 


The  Ratio  of  Serious  to  Non-serious  Errors  as  a  Function  of 
Verbal  Abilities  and  Learning  Stages 


Error 

Types 

—  ■ 

Ability 

Class 

January  Data 

November  Data 

Postman  Stories 

A* 

HMGIII 

64% 

25% 

34% 

91% 

B** 

0^1 

16% 

9<1 

38% 

all 

33% 

1% 

*  Rules  belonging  to  Group  A;  non-serious  errors 

**  Rules  belonging  to  Group  B;  serious  errors 

+  9^1  means  students  whose  Stanford  Vocabulary  Test  scores  are 

in  the  top  15%,  and  9< I  means  the  rest  of  the  students 
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those  students  with  lower  verbal  ability.  Therefore,  it  can  be  said 
that  less  serious  errors  are  observed  among  those  having  higher  verbal 
ability  while  more  serious  errors  were  committed  by  lower  verbal-ability 
students. 

The  November  data  shown  in  Table  8  were  obtained  from  the  same  test 
as  the  test  administered  to  the  seventh  graders  (January  data).  The 
subjects  in  the  November  data  were  given  the  test  after  completion  of 
three  weeks  of  instruction  by  teachers,  and  the  students  performed  very 
well  on  the  test.  But  there  is  no  information  available  on  the  verbal 
abilities  of  these  students.  91%  of  the  observed  rules  in  the  November 
subjects  are  in  Group  A  and  only  1%  belong  to  Group  B.  Erroneous  rules 
of  operation  were  shifted  from  Group  B  to  A  as  the  learning  stages  of 
the  students  advanced.  In  the  January  data,  where  many  students  used  a 
variety  of  rules  consistently,  rules  in  Group  A  were  more  often  used 
among  high  verbal  ability  students. 


DISCUSSION 

Brown  &  Burton  (1978),  K.  Tatsuoka,  et  al.,  (1980)  and  Brown  & 

VanLehn  (1980)  have  found  many  bugs  in  arithmetic  skills  of  whole 
numbers  and  signed  numbers.  Although  their  approaches  toward  diagnosing 
misconceptions  are  not  exactly  the  same,  both  models  require  a  careful 
task  analysis.  Several  researchers  (Airasian,  1971;  Bart  &  Airasian, 

1973;  Dayton,  1976;  Price,  1978)  are  working  toward  constructing 
behavioral  item  hierarchies  by  using  order  analysis.  Sato  (1978,  1980) 
developed  a  method  for  constructing  a  task  tree  by  using  Graph  Theory. 

(His  work  has  been  introduced  in  English  by  M.  Tatsuoka  (1979)).  The 
method  starts  by  making  an  adjacency  matrix  whose  columns  represent 
component  procedures.  After  obtaining  the  reachability  matrix  from  the 
adjacency  matrix,  Sato  constructs  a  network.  With  his  method,  all 
procedural  levels  of  a  task  are  determined  first,  and  then  nodes 
representing  the  steps  of  the  task  are  added  to  each  level.  The 
reachability  matrix  provides  the  direct  and  indirect  relationships  among 
the  nodes  in  the  network. 

The  seriousness  of  errors  is  determined  by  referring  to  the 
procedural  network  (or  task  tree).  If  a  rule  is  the  result  of  a 
misconception  of  an  earlier  level  in  the  network,  then  it  is  more  likely 
committed  by  students  who  are  in  the  early  stage  of  learning,  or  by  lower 
ability  students.  When  students  are  near  the  mastery  level  of 
performance,  their  erroneous  rules  (if  any)  are  due  to  mistakes  occurring  at 
the  end  of  the  network.  Some  erroneous  rules  have  resulted  from 
misconceptions  occurring  at  two  or  more  levels  and  their  sources  of 
errors  are  compounded.  PSCI  was  designed  to  quantitatively  express 
these  compounded  error  sources  as  well  as  those  single  error  sources. 

The  example  in  this  paper  (addition  of  signed  numbers)  has  an 
especially  simple  procedural  representation,  but  our  technique  for 
assessing  the  seriousness  of  errors  should  he  extendable  to  more  general 
cases.  For  example,  the  method  is  now  being  tried  out  on  signed-number 
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subtraction  and  fraction  problems.  Some  networks  might  be  too 
complicated  for  determining  all  the  levels  In  a  linear  order.  If 
several  different  seta  of  linearly  ordered  levels  exist  In  a  procedural 
network,  further  work  will  be  needed  for  generalizing  the  definition  of 
the  Procedural  Steps  Conformity  Index. 
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Army 


1  Technical  Director 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Myron  Fischl 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Dexter  Fletcher 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria  ,VA  22333 

1  Dr.  Michael  Kaplan 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

1  Dr.  Milton  S.  Katz 

Training  Technical  Area 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Harold  F.  O’Neil,  Jr. 

Attn:  PERI-OK 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Mr.  Robert  Ross 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Sasmor 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


1  Commandant 

US  Army  Institute  of  Administration 
Attn:  Dr.  Sherrill 
FT  Benjamin  Harrison,  IN  46256 

1  Dr.  Frederick  Steinheiser 
U.  S.  Army  Reserch  Institute 
5001  Eisenhower  Avenue 
Alexandria.  VA  22333 
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Army 


1  Dr.  Joseph  Ward 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1 

Air  Force 


1  Air  Force  Human  Resources  Lab 
AFHRL/MPD 

Brooks  AFB,  TX  78235 

1  Dr.  Earl  A.  Alluisi 
HQ,  AFHRL  (AFSC) 

Brooks  AFB.  TX  78235 

1  Research  and  Measurraent  Division 
Research  Branch,  AFMPC/MPCYPR 
Randolph  AFB,  TX  78148 

1  Dr.  Malcolm  Ree 
AFHRL/MP 

Brooks  AFB,  TX  78235 

1  Dr.  Marty  Rockway 
Technical  Director 
AFHRL(OT) 

Williams  AFB,  AZ  58224 


Civil  Govt 


1  Dr.  Andrew  R.  Molnar  1 

Science  Education  Dev. 
and  Research 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Vern  W.  Urry  1 

Personnel  RiD  Center 
Office  of  Personnel  Management 
1900  E  Street  HW 
Washington,  DC  20415 

1  Dr,  Joseph  L.  Young,  Director 
Memory  4  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550  ^ 


Marines 


H.  William  Greenup 
Education  Advisor  (E03I) 

Education  Center,  MCDEC 
Quantico,  VA  22134 

Director.  Office  of  Manpower  Utilization 
HQ,  Marine  Corps  (MPU) 

BCB,  Bldg.  2009 
Quantico,  VA  22134 

DR.  A.L.  SLAFKOSKY 
SCIENTIFIC  ADVISOR  (CODE  RD-1) 

HQ.  U.S.  MARINE  CORPS 
WASHINGTON.  DC  20380 

Other  DoD 


Defense  Technical  Information  Center 
Cameron  Station ,  Bldg  5 
Alexandria,  VA  22314 
Attn:  TC 

Dr.  William  Graham 
Testing  Directorate 
MEPCOM/MEPCT-P 
Ft.  Sheridan,  IL  60037 

Military  Assistant  for  Training  and 
Personnel  Technology 

Office  of  the  Under  Secretary  of  Defense 
for  Research  &  Engineering 
Room  3D  129,  The  Pentagon 
Washington,  DC  20301 

MAJOR  Wayne  Sellman,  USAF 
Office  of  the  Assistant  Secretary 
of  Defense  (MRA4L) 

3B930  The  Pentagon 
Washington,  DC  20301 

DARPA 

1400  Wilson  Blvd . 

Arlington,  VA  22209 


CoastGuard 


Mr.  Thomas  A.  Warm 
U.  S.  Coast  Guard  Institute 
P.  0.  Substation  18 
Oklahoma  City,  OK  73169 
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Non  Govt 


1  Charles  Myers  Library 
Livingstone  House 
Livingstone  Road 
Stratford 
London  E15  2LJ 
ENGLAND 

1  Dr.  Kenneth  E.  Clark 

College  of  Arts  &  Sciences 
University  of  Rochester 
River  Campus  Station 
Rochester,  NY  14627 

1  Dr.  Norman  Cliff 
Dept,  of  Psychology 
Univ.  of  So.  California 
University  Park 
Los  Angeles,  CA  90007 

1  Dr.  William  E,  Coffman 

Director,  Iowa  Testing  Programs 
334  Lindquist  Center 
University  of  Iowa 
Iowa  City,  lA  52242 

1  Dr.  Meredith  P.  Crawford 

American  Psychological  Association 
1200  17th  Street,  N.W. 

Washington,  DC  20036 

1  Dr .  Leonard  Feldt 

Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City,  lA  52242 

1  Dr.  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.O.  Box  168 

Iowa  City,  lA  52240 

1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Univ.  Prof.  Dr.  Gerhard  Fischer 
Liebiggasse  5/3 
A  1010  Vienna 
AUSTRIA 

1  DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  15213 


1  Dr,  John  R,  Frederiksen 
Bolt  Beranek  &  Newman 
50  Moulton  Street 
Catibridge,  MA  02138 

1  Dr.  Erling  B.  Andersen 
Department  of  Statistics 
Studiestraede  6 
1455  Copenhagen 
DENMARK 

1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

1  Dr.  Isaac  Be jar 

Educational  Testing  Service 
Princeton,  NJ  08450 

1  Dr.  Werner  Birke 

DezWPs  im  Streitkr aefteamt 
Postfach  20  50  03 
D-5300  Bonn  2 
WEST  GERMANY 

1  Dr.  Nicholas  A.  Bond 
Dept,  of  Psychology 
Sacramento  State  College 
600  Jay  Street 
Sacramento,  CA  95819 

1  Dr.  Robert  Brennan 

American  College  Testing  Programs 

P.  0.  Box  168 

Iowa  City,  lA  52240 

1  DR.  C.  VICTOR  BUNDERSON 
WICAT  INC. 

UNIVERSITY  PLAZA,  SUITE  10 
1160  SO.  STATE  ST. 

OREM,  UT  84057 

1  Dr.  John  B.  Carroll 
Psychometric  Lab 
Univ.  of  No.  Carolina 
Davie  Hall  013A 
Chapel  Hill,  NC  275l4 

1  Library 

HumRRO/Western  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 

1  Dr.  Steven  Hunka 

Department  of  Education 
University  of  Alberta 
Edmonton,  Alberta 
CANADA 
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Non  Govt 


1  Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle.  WA  98105 

1  Dr  .  Huynh  Huynh 

College  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  Professor  John  A.  Keats 
University  of  Newcastle 
AUSTRALIA  2308 

1  Mr.  Marlin  Kroger 
1117  Via  Goleta 

Palos  Verdes  Estates,  CA  90274 

1  Dr.  Charles  Lewis 

Faculteit  Sociale  Wetenschappen 
Ri jksun iverslteit  Groningen 
Oude  Boter ingestraat 
Groningen 
NETHERLANDS 

1  Dr.  Robert  Linn 

College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Frederick  M.  Lord 

Educational  Testing  Service 
Princeton,  NJ  08540 

1  Dr.  Gary  Marco 

Educational  Testing  Service 
Princeton,  NJ  08450 

1  Dr .  Scott  Maxwell 

Department  of  Psychology 
University  of  Houston 
Houston,  TX  77004 

1  Dr,  Samuel  T.  Mayo 

Loyola  University  of  Chicago 
820  North  Michigan  Avenue 
Chicago,  IL  60611 

0 

1  Dr  .  Ron  Hambleton 
School  of  Education 
University  of  Massechusetts 
Amherst,  MA  01002 


1  Dr.  Melvin  R.  Novick 

356  Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City,  lA  52242 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
400  Army  Navy  Drive 
Arlington,  VA  22202 

1  Dr.  James  A.  Paulson 

Portland  State  University 
P.O.  Box  751 
Portland,  OR  97207 

1  MR.  LUIGI  PETRULLO 

2431  N.  EDGEWOOD  STREET 
ARLINGTON,  VA  22207 

1  DR.  DIANE  M.  RAMSEY-KLEE 

R-K  RESEARCH  &  SYSTEM  DESIGN 
3947  RIDGEMONT  DRIVE 
MALIBU,  CA  90265 

1  MINRAT  M.  L.  RAUCH 
P  II  4 

BUNDESMINISTERIUM  DER  VERTEIDIGUNG 

POSTFACH  1328 

D-53  BONN  1,  GERMANY 

1  Dr.  Mark  D.  Reckase 

Educational  Psychology  Dept. 
University  of  Missour i-Colimbia 
4  Hill  Hall 
Colimbia,  MO  65211 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

1  Dr.  Leonard  L.  Rosenbaun,  Chairman 
Department  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Dr.  Chester  Harris 
School  of  Education 
University  of  California 
Santa  Barbara,  CA  93106 

1  Dr.  Lloyd  Humphreys 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 


Ill inois/Tatsuoka  February  23,  1981 
Non  Govt 


1  Dr.  Ernst  Z.  Rothkopf 
Bell  Laboratories 
600  Mountain  Avenue 
Murray  Hill.  NJ  0797‘i 

1  Dr.  Lawrence  Rudner 
403  Elm  Avenue 
Takoma  Park,  MD  20012 

1  Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  PROF.  FUMIKO  SAMEJIMA 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  TENNESSEE 
KNOXVILLE,  TN  37916 

1  DR.  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUMRRO 

300  N.  WASHINGTON  ST. 

ALEXANDRIA,  VA  22314 

1  Dr .  Kazuo  Shigemasu 
University  of  Tohoku 
Department  of  Educational  Psychology 
Kawauchi,  Sendai  980 
JAPAN 

1  Dr.  Edwin  Shirkey 

Department  of  Psychology 
University  of  Central  Florida 
Orlando,  FL  32816 

1  Dr.  Robert  Smith 

Department  of  Computer  Science 

Rutgers  University 

New  Brunswick,  NJ  08903 

1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 


1  Professor  Donald  Fitzgerald 
University  of  New  England 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 
Suite  900 

4330  East  West  Highway 
Washington,  DC  20014 
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1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94305 

1  Dr.  Hariharan  Swaminathan 

Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

1  Dr.  Brad  Sympson 

Psychometric  Research  Group 
Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr.  David  Thissen 

Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

1  Dr.  Robert  Tsutakawa 

Department  of  Statistics 
University  of  Missouri 
Columbia,  MO  65201 

1  Dr.  J.  Uhlaner 

Perceptronics ,  Inc. 

6271  Variel  Avenue 
Woodland  Hills,  CA  91364 

1  Dr.  Howard  Wainer 

Bureau  of  Social  Science  Research 
1990  M  Street,  N.  W. 

Washington,  DC  20036 

1  Dr.  Phyllis  Weaver 

Graduate  School  of  Education 
Harvard  University 
200  Larsen  Hall,  Appian  Way 
Cambridge,  MA  02138 

1  DR.  SUSAN  E.  WHITELY 
PSYCHOLOGY  DEPARTMENT 
UNIVERSITY  OF  KANSAS 
LAWRENCE,  KANSAS  66044 

1  Wolfgang  Wildgrube 
Streitkraefteamt 
Box  20  50  03 


